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ABSTRACT 


Self-driving cars has the potential to revolutionize urban mobility by providing 
sustainable, safe, and convenient and congestion free transportability. 
Autonomous driving vehicles have become a trend in the vehicle industry. 
Many driver assistance systems (DAS) have been presented to support these 
automatic cars. This vehicle autonomy as an application of AI has several 
challenges like infallibly recognizing traffic lights, signs, unclear lane markings, 
pedestrians, etc. These problems can be overcome by using the technological 
development in the fields of Deep Learning, Computer Vision due to 
availability of Graphical Processing Units (GPU) and cloud platform. By using 
deep learning, a deep neural network based model is proposed for reliable 
detection and recognition of traffic lights (TL). 
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INRODUCTION 

The road traffic scenes are usually complex and traffic video 
is vulnerable to external factors such as light, weather and 
obstructions. It is difficult to extract the traffic parameters 
and detect the traffic anomaly exactly with the existing image 
processing and analysis technologies under such uncertainty 
factors. A deep neural network based model is proposed for 
reliable detection and recognition of traffic lights and signs 
using transfer learning. The proposed method incorporates 
use of faster region based convolutional network (R-CNN) 
Inception V2 model in TensorFlow for transfer learning. The 
model was trained on dataset containing different images of 
traffic signals in accordance with Indian Traffic Signals. The 
model accomplishes its objective by detecting the traffic light 
with its correct class type. 

LITERATURE SURVEY 

1. Traffic Lights System Based on RFID for Autonomous 
Driving Vehicle 

Abduladhem Abdulkareem Ali [1] proposed a RFID based 
method used to recognize traffic lights status. The proposed 
system mainly consists of two major block modules the traffic 
lights module and the vehicle module. The traffic lights 
module is made up of two electronic parts, the 
microcontroller and the XBee module. While the vehicle 
module is equipped with the preceding elements in addition 
to the RFID reader. The system is responsible for the 
recognition and proper presentation of those signs to the 
vehicle. 
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The RFID technology uses the electromagnetic fields to 
transmit signals, RFID tags can be placed on the road or 
being embedded into its infrastructure. In this prototype 
system, RFID reader of the vehicle is being placed under the 
vehicle to be in the closest position to detected the tags 
under the road which is shown in figure 1. 
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Figurel. Traffic light recognition system 

Each of the RFID tags should be already defined to both the 
vehicle and the traffic light controller. Once the vehicle passes 
an RFID tag, it broadcasts a request message using the XBee 
module .Only traffic light controller receives that message. 
This message contains two fields: an RFID tag number for the 
vehicle detected and the vehicle ID number. While the vehicle 
ID number defines that address of that vehicle in order to 
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unicast an acknowledge message that contains the traffic light 
status on that road. If the road has multiple track and 
destinations, then all the status of road destinations is 
broadcasted to the vehicle. 

2. Frame based Object Detection - An Application for 
Traffic Monitoring 

Chin Hong Low, Ming Kiat Lee, Siak Wang Khor [2], describe a 
system that is capable of detecting and segmenting objects 
from video frames which helps in traffic surveillance. Shadow 
is one of the problems faced by most of the object detection 
systems. By differentiating the background and foreground of 
the video scenes using MATLAB embedded functions, the 
foreground regions are segmented as the objects. The 
segmented objects are saved forfurther development of 
computer vision applications such as object recognition and 
classification. 

The first step in this method is to convert the video frames to 
images and after that subtracting the background from a 
particular frame. The result of the background subtraction is 
an image that only consists of foreground objects. However, 
this result might be affected by the change of light source, 
image noise and some other factors. To get a better result, a 
threshold should be obtained for the subtracted result. The 
next step is to find a threshold value to convert the 
subtracted image to binary image. The threshold value is a 
midpoint value to segment the foreground and background, a 
range of values are used to differentiate the object and the 
background. The value that is greater than the threshold will 
be taken as an object, otherwise it is decided to be the 
background. The final result will be a binary image where the 
background pixels are set to 0 (black pixels) and the objects 
pixels are set to 1 (white pixels). The result from the previous 
steps may include the shadows of objects in the foreground 
segments. The shadow will affect the accuracy of the object 
detection and segmentation result. Hence in the proposed 
system, the next step is to detect and remove the shadows. 
The method that used to remove shadow is based on HSV 
color space. 

3. Moving object detection and tracking Using 
Convolutional Neural Networks 

Shraddha Mane [3] proposed a novel CNN based object 
tracking algorithm which is used for robust object detection. 
The proposed approach is able to detect the object in 
different illumination and occlusion. The proposed CNN 
based moving object detection algorithm consists of two 
phase: Object detection and tracking. 



Figure 2. Block Diagram of the proposed system 


In this system, the video is taken as an input. Frames are 
extracted for further processing. The two main algorithms 
object detection and object tracking is process through deep 
learning methods therefore Tensor flow based object 
detection algorithm has been used. After object detection, 
their locations are important to start the tracking process. 


Here Convolutional Neural Network (CNN) based tracking 
algorithm is used. In this approach first load the weights of 
the pre-trained model. The model is capable of incorporating 
the temporal information. Rather than focusing on the objects 
in the testing time, the pre-trained model which is trained on 
large variety of objects in real time. This lightweight model 
has ability to track the object at the speed of 150 frames per 
second. Also it is able to remove the barrier of occlusion. In 
this approach, the object locations obtained from the 
TensorFlow based object detection algorithms are passed to 
the CNN based object tracking algorithm. The initial positions 
are learned by the model and the same points are search in 
the net frames by testing process of CNN model. 

4. A Vision Based Traffic Light Detection and 
Recognition Approach for Intelligent Vehicles 

Ziya Ozcelik, Canan Tastimur, Mehmet Karakose, Erhan Akin 
[4] proposed a method which is divided into two sections. 
Each of these parts requires hardware. The first part requires 
a camera to get the image. The other part requires a 
computer to process the received images. In the proposed 
method, images have been taken using CCD camera. Image 
processing techniques are performed step by step to detect 
the traffic lights in the received image through the computer. 
When traffic lights are detected, the received RGB image is 
converted into HSV. By performing a color based 
segmentation process on the obtained HSV format image, the 
locations of traffic lights in the image are easily detected. 
The color of the traffic light is easily determined through the 
SVM (Support Vector Machines) classification model, which is 
a machine learning algorithm. After the detection of the 
objects (circle) on the image, the detection of the states of the 
traffic lights detected is performed. Machine learning is used 
to determine the state of the traffic light therefore they have 
been categorized into three traffic lights (Red, Yellow and 
Green).Here, the number of groups to be classified is three 
because of the traffic light conditions. In order to create the 
SVM model, training is carried out using approximately 8000 
frames of image obtained in the traffic. During training is 
characterized by a 3D color histogram method, which 
contains 8 boxes per channel respectively. After 
characterization, the method returns a feature vector of size 
D = 8x8x8 = 512. The obtained feature vector is used for 
modeling. For each image, the SVM model is created by 
performing the mentioned method. After the SVM model has 
been created, the model obtained is combined with image 
fragment within the boundaries of the object (circle) on the 
image to check the traffic light status. 

5. Traffic Light Detection and Recognition based on 
Haar-like Features 

Sang-Hyuk Lee, Jung-Hawn Kim, Yong-Jin Lim and Joonhong 
Lim [5] proposed an algorithm for traffic light detection and 
recognition based on Haar-like features. Here Haar-like 
features are used to learn about the traffic light image and 
detect the candidate area based on the learning data. The 
detected candidate image is verified by the pre-learned SVM 
(Support Vector Machine) classifier, and binarization and 
morphology operations are performed on the verified 
candidate image for detection of the traffic light object. The 
detected traffic light is divided into respective signal areas to 
determine the current on/off status of traffic lights. The 
signal signs in the respective areas are defined by regulation 
and the sign of traffic lights can be recognized by recognizing 
on/off of the signals in the respective areas. 
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6. Traffic Light Detection During Day and Night 
Conditions by a Camera 

Chunhe Yu, Chuan Huang, Yao Lang [6] create a method to 
detect traffic light in day and night condition real time using 
camera., the algorithm of traffic lights detection which is 
applied in a vehicle driver assistance system is designed by 
using the image processing technology. The system of traffic 
light detection includes three parts: a CCD camera, an image 
acquisition card, and a PC. 

The proposed method includes three steps: object extraction 
by color, rule based traffic light verification, and then extract 
the signal of the traffic light. The color segmentation is 
performed using RGB color model. They use color model 
element range and threshold to determine traffic light 
candidates. Red traffic light candidate must have R element 
bigger than G and B with a certain threshold, green traffic 
light candidate must have G element bigger than R and B 
with a certain threshold, and yellow traffic light must have R 
and G element bigger than B component with a certain 
threshold. Based on RGB color space, the algorithm extracts 
red, green, and yellow objects in the image firstly; For the 
purpose of eliminating disturbance in the environment, the 
features of traffic lights are used to verify the object identity, 
and then the types of traffic signals are judged. 

7. Traffic Light and Sign Detection for Autonomous 
Land Vehicle Using Raspberry Pi 

Priyanka D, Dharani K, Anirudh C, Akshay K [7] proposed a 
new method based on Raspberry Pi. Automatic recognition of 
traffic signs is also important for an automated intelligent 
driving vehicle or for a driver assistance system. This is a 
visual based project i.e., the input to the system is video data 
which is continuously captured from the webcam is 
interfaced to the Raspberry Pi. Images are pre-processed 
with several image processing techniques such as; Hue, 
Saturation and Value (HSV] color space model technique is 
employed for traffic light detection, for sign detection again 
HSV color space model and Contour Algorithm has been used. 
The signs are detected based on Region of Interest (ROI]. The 
ROI is detected based on the features like geometric shape 
and color of the object in the image containing the traffic 
signs. 


Figure3. Proposed Block Diagram 

In the proposed work, Raspberry Pi 3 is the heart of the 
system. Raspberry Pi 3 is a compact computer which has 
many ports to which Camera, Speaker, etc., is interfaced. With 
the help of Raspberry Pi, the Image processing is done 
through OpenCV using Python. The maximum resolution of 
the Webcam is 1280x720, in which the resolution of the 
processed image is 680x680 and the Line of Sight (LOS] is 
600 - 1200. The operating range of the camera is 40cm to 


130cm and with Raspberry Pi camera the range is 50cm to 
400cm. When a sign or traffic light is detected audio 
command is played through the speaker. 

8. Segment-Based Traffic Sign Detection from Mobile 
Laser Scanning Data 

Ying Li, Lingfei Ma, Yuchun Huang [8] proposes a segment- 
based traffic sign detection method using vehicle-borne 
mobile laser scanning (MLS] data. This method has three 
steps: road scene segmentation, clustering and traffic sign 
detection. The non-ground points are firstly segmented from 
raw MLS data by estimating road ranges based on vehicle 
trajectory and geometric features of roads (e.g., surface 
normal and planarity]. The ground points are then removed 
followed by obtaining non-ground points where traffic signs 
are contained. Secondly, clustering is conducted to detect the 
traffic sign segments (or candidates] from the non-ground 
points. Finally, these segments are classified to specified 
classes. Shape, elevation, intensity, 2D and 3D geometric and 
structural features of traffic sign patches are learned by the 
support vector machine (SVM] algorithm to detect traffic 
signs among segments. The proposed algorithm has been 
tested on a MLS point cloud dataset acquired by a Leador 
system in the urban environment. The results demonstrate 
the applicability of the proposed algorithm for detecting 
traffic signs in MLS point clouds. The proposed algorithm is 
illustrated in the figure given below. 



Figure4. Flowchart of the proposed traffic sign 
detection 

9. Traffic Sign Detection and Classification using Colour 
Feature and Neural Network 

Md. Abdul Alim Sheikh [9] proposed a new method based on 
neural network. The algorithm contain three main stages: 1] 
Preprocessing 2] Detection 3] Recognition and Classification. 
The system takes static color images as input taken from 
World Wide Web. The main steps of the proposed 
methodology are shown in Fig. 5. The output of the system is 
the name of the recognized road sign. In the detection 
module, the region of interest from the image is extracted 
based on color segmentation and make them ready for the 
recognition stage. The second module, artificial neural 
network is used to carry out the recognition and classification 
job. 
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Figure5. Stages of the Proposed Method 

10. Traffic Light Color Identification for Automatic 
Traffic Light Violation Detection System 

Pasit Wonghabut, Rujchai Ung-arunyawee, Watis Leelapatra 
[10] proposes a software technique based on HSV (Hue, 
Saturation, Value] color model to eliminate difficulties in red 
light detection and is able to identify all colors of traffic light. 
Automatic traffic light violation detection system relies on 
color detection of traffic light appeared in video frames. The 
proposed system mainly includes three methods. 

Method 1: Convert video frames to grayscale image, then 
followed by thresholding which yields binary images. All 
three colors of traffic light are compared with predefined 
pattern to determine which one is on. If there are more than 
one color of traffic light found to be on, the threshold value is 
adjusted and the process is repeated and this method uses 
pixel threshold value of ROI to judge whether the lights are 
switched on or not. 

Method 2: Convert video frames from RGB to HSV color 
system, then compare value of hue to determine color of 
traffic light this method utilizes HSV color model to analyze 
ROI. 

Method 3: Combine result of above methods. The color of 
traffic light is produced when the results from two methods 
are identical. 

Three methods are combined to identify traffic light color. 
The conditional analysis is a process used to handle 
ambiguity of traffic light using predefined pattern as shown 
in Fig.4. This process makes decision whether the algorithms 
should output red, yellow or green. To verify correctness of 
algorithm, the number of times each color of traffic light 
switched on is counted manually by human and use this 
result as reference. 



Figure6. Traffic lights decision pattern 


CONCLUSION 

Various methods to detect and recognize the traffic lights 
and signs are analyzed. Each of them works with different 
schemes. Each scheme varies on the convolution layers used 
and also their application used. A Deep learning system that 
classifies the traffic light is considered. The system consists 
of an ensemble of CNNs with post processing neural network 
that combine the predictions from the CNNs with traffic 
lights metadata. In the field of self-driving cars or 
autonomous driving, the use of Faster R-CNN Inception-V2 
model via transfer learning improves the accuracy which 
makes the system reliable for real time application 
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